Generally, personal computers (PCs) and other types of computer systems have been designed around a shared bus system for accessing memory. One or more processors and one or more input/output (I/O) devices are coupled to memory through the shared bus. The I/O devices may be coupled to the shared bus through an I/O bridge which manages the transfer of information between the shared bus and the I/O devices, while processors are typically coupled directly to the shared bus or are coupled through a cache hierarchy to the shared bus.
Unfortunately, shared bus systems suffer from several drawbacks. For example, the multiple devices attached to the shared bus present a relatively large electrical capacitance to devices driving signals on the bus. In addition, the multiple attach points on the shared bus produce signal reflections at high signal frequencies which reduce signal integrity. As a result, signal frequencies on the bus are generally kept relatively low in order to maintain signal integrity at an acceptable level. The relatively low signal frequencies reduce signal bandwidth, limiting the performance of devices attached to the bus.
Lack of scalability to larger numbers of devices is another disadvantage of shared bus systems. The available bandwidth of a shared bus is substantially fixed (and may decrease if adding additional devices causes a reduction in signal frequencies upon the bus). Once the bandwidth requirements of the devices attached to the bus (either directly or indirectly) exceeds the available bandwidth of the bus, devices will frequently be stalled when attempting access to the bus, and overall performance of the computer system including the shared bus will most likely be reduced.
On the other hand, distributed memory systems lack many of the above disadvantages. A computer system with a distributed memory system includes multiple nodes, two or more of which are coupled to different memories. The nodes are coupled to one another using any suitable interconnect. For example, each node may be coupled to each other node using dedicated lines. Alternatively, each node may connect to a fixed number of other nodes, and transactions may be routed from a first node to a second node to which the first node is not directly connected via one or more intermediate nodes. A memory address space of the computer system is assigned across the memories in each node.
In general, a “node” is a device which is capable of participating in transactions upon the interconnect. For example, the interconnect may be packet-based, and the node may be configured to receive and transmit packets as part of a transaction. Generally speaking, a transaction is a series of packets. A “requester” or “source” node initiates a transaction directed to a “target” node by issuing a request packet. Each packet, which is part of the transaction, is communicated between two nodes, with the receiving node being designated as the “destination” of the individual packet. When a packet ultimately reaches the target node, the target node accepts the information conveyed by the packet and processes the information internally. Alternatively, a node located on a communication path between the source and target nodes may relay the packet from the requester node to the target node.
In addition to the original request packet, the transaction may result in the issuance of other types of packets, such as responses, probes, and broadcasts, each of which is directed to a particular destination. For example, upon receipt of the original request packet, the target node may issue broadcast or probe packets to other nodes in the processing system. These nodes, in turn, may generate responses, which may be directed to either the target node or the requester node. If directed to the target node, the target node may respond by issuing a response back to the requester node.
Distributed memory systems present design challenges which differ from the challenges in shared bus systems. For example, shared bus systems regulate the initiation of transactions through bus arbitration. Accordingly, a fair arbitration algorithm allows each bus participant the opportunity to initiate transactions. The order of transactions on the bus may represent the order that transactions are performed (e.g. for coherency purposes). On the other hand, in distributed systems, nodes may initiate transactions concurrently and use the interconnect to transmit the transactions to other nodes. These transactions may have logical conflicts between them (e.g. coherency conflicts for transactions involving the same address) and may experience resource conflicts (e.g. buffer space may not be available in various nodes) since no central mechanism for regulating the initiation of transactions is provided. Accordingly, it is more difficult to ensure that information continues to propagate among the nodes smoothly and that deadlock situations (in which no transactions are completed due to conflicts between the transactions) are avoided.
For example, certain deadlock conditions may occur in known I/O systems, such as the Peripheral Component Interconnect (PCI) I/O system, unless packets associated with a “posted” write transaction are allowed to pass other traffic not associated with a posted write transaction. Generally speaking, a posted write transaction is considered complete by the requester when the write request and corresponding data are transmitted by the requester (e.g., via a source interface), and thus, is effectively completed at the requester. Because the requester is not directly aware of when the posted write transaction is actually completed by the target, more ordering support for the posted operations must be provided in hardware. Thus, the requester may issue additional requests while the packet or packets of the posted write transaction travel to the target, with the assumption that such additional requests will complete after the initial posted transaction completes. Sufficient hardware must be available to support this assumption.
In contrast, a “non-posted” write transaction is not considered complete by the requester until the target (e.g., a target interface) has completed the non-posted write transaction. The target generally transmits an acknowledgement to the requester when the non-posted write transaction is completed. Such acknowledgements consume interconnect bandwidth and must be received and accounted for by the requester. Non-posted write transaction may be issued, for example, when the requester needs to know that previous transactions have been completed before following ones are issued.
In a computer system having a distributed memory system, memory requests (e.g., read and write operations) originating from I/O nodes may need to be properly ordered with respect to other pending memory operation to preserve memory coherency within the computer system and to satisfy any ordering requirements of the I/O system. For example, memory operations may need to be completed in the order in which they were generated to preserve memory coherency within the computer system and to satisfy I/O ordering requirements. It would thus be desirable to have a computer system implementing a system and method for providing a separate communication channel for posted requests. Such a system and method would avoid deadlock situations, while also minimizing the apparatus (in terms of hardware) to enhance ease of implementation.